Automated motion data processing

ABSTRACT

Motion data processing includes receiving a plurality of motion data samples, wherein the plurality of motion data samples comprises a plurality of acceleration data samples obtained by an accelerometer and a plurality of angular velocity data samples obtained by a gyroscope, and the plurality of motion data samples pertains to a plurality of motions performed by a user; and correcting for distortions in the motion data samples, including determining, based at least in part on some of the motion data samples, a set of one or more corrected core measurements of the user&#39;s body positions during a subset of the plurality of motions.

CROSS REFERENCE TO OTHER APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationSer. No. 14/464,250, entitled AUTOMATED MOTION DATA PROCESSING filedAug. 20, 2014 which is incorporated herein by reference for allpurposes, which claims priority to U.S. Provisional Application No.61/868,865, entitled HARDWARE DEVICES AND THE ASSOCIATED SOFTWAREAPPLICATIONS TO SENSE, CAPTURE, ANALYZE, TRACK AND CONNECT REAL-TIMEWORKOUT DATA AND CONTEXTS TO HELP PEOPLE ACHIEVE WORKOUT GOALS filedAug. 22, 2013 which is incorporated herein by reference for allpurposes.

U.S. patent application Ser. No. 14/464,250, entitled AUTOMATED MOTIONDATA PROCESSING filed Aug. 20, 2014 claims priority to U.S. ProvisionalApplication No. 61/940,492, entitled USER INTERACTIONS, ALGORITHMS ANDINDUSTRIAL DESIGN TO ADOPT WEARABLE MOTION SENSING DEVICES IN FITNESSEXERCISES AND SPORTS filed Feb. 17, 2014 which is incorporated herein byreference for all purposes.

BACKGROUND OF THE INVENTION

Conventional ways of exercising to achieve fitness goals can bechallenging for many people. Exercising typically involves performingrepetitive motions, which many view as tedious and therefore notenjoyable. Moreover, it is difficult for individuals to track theperformance of their exercises besides using basic measurements such astime and distance. Today, one often needs to hire a personal trainer toget deeper knowledge and advice about one's workouts. In addition to thehigh cost, the expertise of the personal trainers can vary and thequality of their advice cannot be guaranteed.

There are some wearable devices that provide functions such as automaticstep counting or distance tracking. These devices, however, typicallyonly provide limited information about basic exercises such as how mucha person has walked in a day, which can be insufficient for those whowant to improve the quality of their workouts in addition to quantity.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1 is a system diagram illustrating an embodiment of an automatedfitness coaching system.

FIG. 2A is a functional diagram illustrating an embodiment of anautomated fitness coaching system.

FIG. 2B is a block diagram illustrating an embodiment of a sensordevice.

FIG. 3 is a flowchart illustrating an embodiment of a process performedby the automated fitness coaching system.

FIG. 4 is a block diagram illustrating an embodiment of an evaluationmodule.

FIG. 5A is a flowchart illustrating an embodiment of an improved processfor transforming the sensors' outputs and determining form measurements.

FIG. 5B is a diagram illustrating an example statistical model of groundtruth velocities relative to a specific sensor output feature in asingle dimension.

FIG. 6 is a flowchart illustrating an embodiment of a process fordetermining a form measurement associated with a single repetition of anexercise motion.

FIG. 7A is a sample data diagram illustrating the data processing ofFIG. 6.

FIG. 7B is a diagram illustrating a simulation of a user's leg motionduring a running exercise.

FIG. 8 is an example user interface screen displaying a screenshot of acoached boxing workout.

FIGS. 9A-9B are example user interface screens of showing feedback.

FIGS. 10A-10B are block diagrams illustrating a learning machine.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess; an apparatus; a system; a composition of matter; a computerprogram product embodied on a computer readable storage medium; and/or aprocessor, such as a processor configured to execute instructions storedon and/or provided by a memory coupled to the processor. In thisspecification, these implementations, or any other form that theinvention may take, may be referred to as techniques. In general, theorder of the steps of disclosed processes may be altered within thescope of the invention. Unless stated otherwise, a component such as aprocessor or a memory described as being configured to perform a taskmay be implemented as a general component that is temporarily configuredto perform the task at a given time or a specific component that ismanufactured to perform the task. As used herein, the term ‘processor’refers to one or more devices, circuits, and/or processing coresconfigured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

An automated fitness coaching system is disclosed. When a user of theautomated fitness coaching system performs a specific exercise, hetypically performs a series of repetitive motions. The automated fitnesscoaching system determines one or more corrected core measurementsassociated with a subset of the motions based on acceleration datasamples, velocity data samples, and optionally magnetic field datasamples that are measured by sensors. In some embodiments, one or moreform measurements are further determined based on the core measurements.In some embodiments, feedbacks based on evaluation results are providedto the user.

FIG. 1 is a system diagram illustrating an embodiment of an automatedfitness coaching system. In this example, one or more sensor devices102-106 are used to detect motions performed by a user. A user wears oneor more sensor device(s) during exercise (e.g., on the ankle whilerunning, on arms and legs while swimming, etc.) to capture motion data.As will be described in greater detail below, a sensor device includesone or more accelerometers and one or more gyroscopes, and optionally,one or more magnetometers are configured to take measurement datasamples of acceleration, angular velocity, and optionally magneticfield, respectively, in connection with the user motions. In someembodiments, the accelerometer(s), the gyroscope(s), and themagnetometer(s) are implemented using one or more integrated circuits(ICs).

The data samples are processed to evaluate the user's performance andoptionally provide feedback. In some embodiments, a sensor deviceincludes an onboard processor configured to perform the data processingand evaluation, as well as an output mechanism to provide feedback(e.g., a speaker to provide audio feedback, a video screen or a light toprovide visual feedback, a haptic display to provide tactile feedback,etc.) such as information about the user's form, the difference betweenthe user's form and the target, what the user can do to improve hisform, etc. In some embodiments such as the example shown, the sensordevices are connected (e.g., via a wireless communication interface suchas Bluetooth interface) to a management device 110, which can be apersonal computer, a smartphone, a tablet, or any other appropriatecomputing device that is configured to perform the data processing,evaluation, and/or feedback. The feedback information can be provided tothe user on the management device, provided to the sensor device to beplayed, or otherwise conveyed to the user. The management device alsoexecutes configuration and any other related management application tomanage the configuration and operations of the sensor devices.

In some embodiments, the management device (or the sensor device if thesensor device directly performs processing) is optionally connected to aserver 120 via a network such as the Internet. Data such asconfiguration, measurements, performance, etc. can be stored on theserver to be accessed later and/or further processed.

FIG. 2A is a functional diagram illustrating an embodiment of anautomated fitness coaching system. System 200 includes a communicationinterface 202 configured to receive motion data samples pertaining tomotions performed by a user of the system. Examples of a communicationinterface include without limitation external connections, such as aport, cable, wireline or wireless network interface card, etc., andinternal connections such as a communication bus.

System 200 further includes an evaluation module 204 coupled to thecommunication interface. The data samples received from the sensors tendto be inaccurate. Thus, the evaluation module is configured todetermine, based at least in part on the motion data samples, one ormore corrected core measurements such as velocity and orientationassociated with a subset of the user's motions. In some embodiments, theevaluation module also derives, based on the corrected coremeasurements, form measurements of the user's body positions for asubset of the user's motions. The evaluation module can be implementedas software components executing on one or more processors, as hardwaresuch as programmable logic devices and/or Application SpecificIntegrated Circuits designed to perform certain functions, or acombination thereof. In some embodiments, the module can be embodied bya form of software products which can be stored in a nonvolatile storagemedium (such as optical disk, flash storage device, mobile hard disk,etc.), including a number of instructions for making a computer device(such as personal computers, servers, network equipment, etc.) implementthe methods described in the embodiments of the present application.

System 200 further includes a feedback module 206 coupled to theevaluation module. Based on the evaluation results, the feedback moduleprovides feedback to the user via audio, visual, and any otherappropriate form of output. In various embodiments, the feedback moduleincludes a microphone, a display screen, an LED, a haptic output thatprovides tactile feedback, and/or any other appropriate output device.

The modules may be implemented on a single device or distributed acrossmultiple devices. The functions of the modules may be included inanother module or further split into multiple sub-modules. In someembodiments, the communication interface, the evaluation module, and thefeedback module are all implemented on a management device, where themotion data samples are sent by a sensor device to the managementdevice, received by its communication interface, evaluated by theevaluation module, and feedback information is presented by the feedbackmodule. In some embodiments, the communication interface, the evaluationmodule, and the feedback module are all implemented on a sensor device,where the motion data samples are captured by sensors on the sensordevice, sent via the communication interface to the evaluation module onthe sensor device to be evaluated, and feedback information is providedby the feedback module on the sensor device. In some embodiments, thecommunication interface and the modules (or subcomponents of themodules) are implemented on separate devices. For example, theevaluation of form information can be performed on the sensor device,and the processed information is sent to the management device forfeedback and display.

FIG. 2B is a block diagram illustrating an embodiment of a sensordevice. In this example, sensor device 200 is worn by a user (e.g., onhis ankle or on his arm) while he exercises. Sensor device 200 includesan inertial magnetic unit (IMU) which includes a 3-axis accelerometer211, a 3-axis gyroscope 212, and an optional 3-axis magnetometer 213that are configured to take sample measurements of the user's motionduring the exercise. A single repetition of a motion (e.g., a singlestep during a running exercise, a single punch during a boxing exercise,a single stroke during a swimming exercise) may comprise N accelerationsamples measured by the accelerometer, N angular velocity samplesmeasured by the gyroscope, and N magnetic field samples measured by themagnetometer (where N is an integer value). In some embodiments, the IMUis implemented using a chipset provided by an electronics manufacturersuch as InvenSense.

In this example, sensor device 200 optionally includes a microcontrollerunit 230, with which IMU 210 is connected. In this example, samplemeasurement data collected by the IMU is offloaded to a managementdevice to be further processed. MCU 230 sends and receives data gatheredby the IMU through a communication interface 220, which can be aBluetooth LE or any other appropriate wireless connection, with amanagement device 110 such as a smart phone, tablet or PC. In someembodiments, other sensor device designs can be employed. For example,the sensor device can include an onboard processor coupled to the IMUand configured to process the sample measurement data and evaluate thecorrected core measurements and/or form measurements locally.

Sensor device 200 optionally includes an RGB LED 260 driven by LEDdriver 251. The light color and strength of each LED in the RGB LED arecontrolled by the sensor device itself or the management device, and thecolor and strength values are set according to the LED manufacturer'sspecifications. In some embodiments, the LED provides feedbackinformation. For example, when the user is exercising with good formthat meets a target, the LED emits green light; when the user is notmeeting the target, the LED emits red light. The sensor device alsooptionally includes a button 240. In various embodiments, the button ismultifunctional and can be used as power on a switch/sleep modeactivator or a general purpose user interface button. The sensor devicealso optionally includes a light sensor and white LED 290, which isconfigured to detect different types of mounting chassis on which thedevice is mounted. The sensor device 200 also includes a battery 280.The voltage delivered to the other parts of the sensor device isdelivered through power management module 270.

FIG. 3 is a flowchart illustrating an embodiment of a process performedby the automated fitness coaching system. Depending on implementation,process 300 can be performed on the sensor device and/or on themanagement device.

At 302, a plurality of motion data samples is received. The motion datasamples pertain to a set of motions performed by a user. The motion datasamples include a set of acceleration data samples taken by theaccelerometer, a set of angular velocity data samples taken by thegyroscope, and optionally a set of magnetic field data samples taken bythe magnetometer. In other words, the samples are taken while a userwearing one or more sensor devices is performing a set of exercisemotions. For example, while running, the user wears a sensor device onhis leg to capture motion data of the leg's movements; while swimming,the user wears a sensor on his arm to capture motion data of the strokesmade by the arm, and a sensor on his leg to capture motion data of thekicks made by the leg. Additional sensor devices can be worn so thatmotion data of both sides of the body is captured.

In some embodiments, the sensor device includes a processor thatperforms the data processing, and the data is received via an internalcommunication interface (e.g., a bus). In some embodiments, themanagement device includes a processor that performs the dataprocessing, and the data is received at the management device via anexternal communication interface (e.g., a Bluetooth LE or other types ofwireless connection).

At 304, one or more corrected core measurements associated with theuser's form for a subset of the motions is evaluated based at least inpart on some of the motion data samples. Specifically, the correctedcore measurements measure the user's body positions while he performsthe subset of motions. In some embodiments, the corrected coremeasurements include a velocity (expressed as a vector V) and anorientation (also referred to as a quaternion, and expressed as a vectorQ). Other corrected core measurements can be determined. Details of theevaluation are described below.

Since exercise motions are often repetitive, in some cases eachcorrected core measurement measures the body positions of the userduring a single repetition of the exercise motions (e.g., a single stepduring running, a single stroke during swimming, a single punch duringboxing, etc.). Optionally, one or more form measurements measuring theuser's form during the single repetition are derived based on the coremeasurement data. Examples of form measurements include cadence, stride,range of motion, efficiency, speed, landing impact, pronation, etc. Theform measurements can be instantaneous measurements of a single motionrather than averages of a series of motions. For example, the user'sinstantaneous stride is determined by measuring how much time it takesfor him to run a single step, and the cadence is the reciprocal of thestride; the user's instantaneous range of motion is determined bymeasuring the maximum angular motion of his leg between the most forwardand most backward positions; the user's instantaneous efficiency isdetermined based on the energy concentration of the angular velocity;the user's instantaneous speed is determined based on velocity; theuser's landing impact is determined based on acceleration; the user'spronation is determined based on the orientation, and so on. Asdiscussed above, the evaluation can be performed by the sensor device orby the management device. In some embodiments, multiple formmeasurements are used to further derive other form information, such asthe average, mean, maximum, minimum, standard deviation, etc. In someembodiments, the corrected core measurements are used directly as formmeasurements. For example, during a boxing exercise, the velocity andorientation measurements are used to measure the user's form indelivering a punch. Details of how to evaluate a measurement using atleast some of the set of motion data samples are discussed below.

At 306, feedback is provided based on an evaluation result.

In some embodiments, the corrected core measurements obtained by theevaluation module are directly compared with certain targets. Forexample, the velocity and rotation of a punch motion performed by theuser are compared with those of a pre-recorded punch motion performed bya coach. In some embodiments, the derived form measurements are comparedwith a target. For example, the cadence, range of motion, etc., can becompared with pre-specified target goals. In some embodiments, thetargets are specified by the user manually or by the systemautomatically. In some embodiments, the targets are specified for aparticular set of exercises and/or as a part of a coached workoutprogram. Feedback is provided to the user based at least in part on thecomparison. For example, the user may be told and/or shown how much heis off target, and be given tips on how to adjust the movements toachieve target.

In some embodiments, a trained feedback model is used to intelligentlyprovide feedback based on the evaluation result. Details of the feedbackmodel are described below.

The following is an example technique of evaluating a set of correctedcore measurements of the user's body positions during a subset ofmotions using the motion data samples obtained from sensors.

In this example, each input to the evaluation module includes a set ofsensor output sample data in an object space, including: anaccelerometer output that measures acceleration and is represented asAx, Ay, Az (or a vector A for short), the units of the accelerationvector being in m/s² or in gravity units; a gyroscope output thatmeasures the angular velocity and is represented as Gx, Gy, Gz (or avector G for short), the units of the angular velocity vector being inrad/s or degree/s; and an optional magnetometer output that measures themagnetic field and is represented as Mx, My, Mz (or a vector M forshort), the units of the magnetic field vector being in Gauss or Tesla.

The reference space of the measurements is a world space coordinatesystem. Specifically, the world space coordinate system is representedas Xw, Yw, Zw, where the Zw axis (0, 0, 1) corresponds to gravitydirection; the Xw axis (1, 0, 0) corresponds to North direction (i.e.,the magnetic north on a horizontal plane); and the Yw axis (0, 1, 0) isthe cross product of Zw and Xw.

As will be described in greater detail below, an evaluation module suchas 400 of FIG. 4 performs a process such as process 500 of FIG. 5A toobtain one or more of the following outputs: a velocity in world space,represented as Vx, Vy, Vz (or a vector V for short); an acceleration inworld space, represented as (Ax, Ay, Az)w (or a vector Aw for short); arotational velocity in world space and local space, represented as (Rx,Ry, Rz)w and (Rx, Ry, Rz)o, respectively (or vectors Rw, Ro for short);a quaternion (the quotient of two vectors) describing how to transformfrom a point in the local space to world space (e.g., the orientationand rotation) and another quaternion representing the world space tolocal space transformation, represented as (Qx Qy Qz Qw)o->w (or vectorQ for short) and (Qx Qy Qz Qw)w->o (or vector Qw for short),respectively; or the relative position in world space, represented as(Px, Py, Pz) (or vector P for short).

In some embodiments, the corrected core measurements include thevelocity vector V and the quaternion vector Q, and are processed further(e.g., by higher level software applications) to give the user feedback,coaching tips, etc. Given V and Q, other outputs can be computed. Forexample the relative position can be computed as P(n)=V(n)*time_step; Qwis the inverse of Q; Rw=Ro*Q.

Although there are standard ways of computing the outputs based on thesensor sample data, in practice, however, the original sensor readingsA, G, and M are highly noisy. Factors contributing to the noise includebiased reading, nonlinearities and mismatching of components, etc. Usingthe standard ways to calculate the output will lead to significanterrors in the result. An error correcting estimation technique isdisclosed in connection with FIGS. 4 and 5.

FIG. 4 is a block diagram illustrating an embodiment of an evaluationmodule. As shown, evaluation module 400 includes a preprocessor 401 thattranscodes data received on the interface to normalized space, alignsthe axis, and performs initial calibrations on the accelerometer,gyroscope, and magnetometer measurements. Conventional techniques fortranscoding, alignment, and calibration can be used. In someembodiments, the preprocessor further performs interpolation on receivedsamples to estimate any sample data that may have been lost or corruptedduring transmission.

The preprocessed data samples are sent to an orientation estimator 402to estimate an interpolated and corrected current orientation, Q(n). Inthis example, the orientation estimation module includes an uncorrectedorientation estimator 404 that determines an uncorrected currentorientation of the current sample, Q′(n). An orientation correctionmodule 406 makes a correction of Q′(n) based on the calibratedaccelerometer output and optionally the magnetometer output, andgenerates a corrected current orientation Q″(n). An interpolator 408interpolates the current sample Q′(n) and the corrected currentorientation Q″(n) to generate an interpolated current orientation Q(n).

The output of the orientation estimator is sent to a velocity estimator410 to estimate a current velocity. In this example, velocity estimator410 includes an acceleration transformer 412 that transforms theacceleration reading in world space based on the estimated orientation,and removes the gravity vector to determine an absolute acceleration inworld space. Velocity estimator 410 further includes an integrator 414that determines an uncorrected velocity based on integrating velocitysamples and acceleration. A statistical model 416 is applied to theoutput of the integrator to generate an error-corrected velocityestimation.

FIG. 5A is a flowchart illustrating an embodiment of an improved processfor transforming the sensors' outputs and determining form measurements.Process 500 corrects for errors in the raw sensor outputs to generatemore accurate form information. Process 500 can be performed on thesensor device itself and/or on the management device, and is used toimplement 304 of FIG. 3 in some embodiments.

In some embodiments, the sensors are calibrated and initialized by apreprocessor. For example, the sensors can have offsets due toimperfections in their hardware. An initial calibration process adjustsfor such offsets and errors. In some embodiments, the initialcalibration process is performed by the manufacturer of the sensordevice. A set of calibration functions with certain calibrationparameters is applied to the raw sensor outputs A, G, and M to obtaincalibrated and initialized outputs Ai, Gi, and Mi as follows:

Ai=calibration_(Ai)(A,calibration_parameters)  (1)

Gi=calibration_(Gi)(G,calibration_parameters)  (2)

Mi=calibration_(Mi)(M,calibration_parameters)  (3)

The actual calibration functions and calibration parameters aredetermined empirically and may vary in different implementations. Forexample, the calibration function can subtract certain offsets from araw output, multiply the raw output by a constant, etc.

Corrected orientation data is determined based at least in part on theangular velocity data, the acceleration data, and the magnetic fielddata if available. Preferably, preprocessed data based on the rawmeasurements is used. At 502, the current orientation (of the currentsample n) is determined based at least in part on the angular velocitydata samples (gyroscope reading Gi), which has a linear relationship tolocal space rotational velocity Ro. In this example, Gi is the same asRo, although other linear relationships may exist between the two inother embodiments. Specifically, the current orientation data sample(expressed as the quaternion Q′(n), n being the sample index or timeindex) is updated by performing a quaternion transform operation on theprevious interpolated orientation data sample Q(n−1) and the currentlocal space rotational velocity Ro(n) as follows:

Q′(n)=transform_(Q)(Q(n−1),Ro(n),time_step)  (4)

where transform_(Q) is a standard quaternion transform function.

If left uncorrected, the output Q′(n) will drift over time. Thus, at504, a corrected orientation Q″(n) is determined based at least in parton the acceleration data. Particular, the preprocessed acceleration dataAi and the preprocessed magnetic field data Mi are used in thedetermination. Specifically, Ai and Mi form a local coordinate systemrelative to the world coordinate system. The local coordinate system isa coordinate system that is relative to the device itself. Therotational transform of the world space to the local coordinate systemis expressed as Q″(n) and derived as follows:

First, a 3×3 rotational matrix Mr is constructed from Ai and Mi, where

vec3Y=normalize(Vec3::Cross(Mi,Ai));  (5a)

vec3X=normalize(Mi);  (5b)

vec3Z=normalize(Ai);  (5c)

Mr.column1.x=X.x;  (5d)

Mr.column1.y=Y.x;  (5e)

Mr.column1.z=Z.x;  (5f)

Mr.column2.x=X.y;  (5g)

Mr.column2.y=Y.y;  (5h)

Mr.column2.z=Z.y;  (5i)

Mr.column3.x=X.z;  (5j)

Mr.column3.y=Y.z;  (5k)

Mr.column3.z=Z.z;  (5l)

Then, rotational matrix Mr is then converted to a quaternion Q″(n) usinga standard mathematical transform.

At 506, an interpolated and corrected current orientation Q(n) isdetermined by interpolating Q′(n) and Q″(n). Any known interpolationfunction can be used. In some embodiments, a weighted linearinterpolation function is used as follows:

Q(n)=interpolate(Q′(n),Q″(n),alpha)  (6)

Typically, 0<alpha<1. The weight alpha is based on the stableness of theraw sensor reading. The more stable the reading is, the greater weightis assigned to Q″(n). In some embodiments, the stability of the sensorreading is determined based on a probability distribution (discussed ingreater detail below in connection with step 512). A more concentratedprobability distribution indicates greater stability and corresponds toa greater weight.

The corrected velocity is determined based at least in part on Q(n) anda statistical model. At 508, an acceleration in world space, Aw(n), isdetermined. Since the accelerometer reading includes the earth'sgravity, the transform function transforms the acceleration reading inworld space Ai(n) based on the estimated orientation, and then removesgravity from the transformed value to determine the acceleration inworld space as follows:

Aw(n)=transform_(v)(Ai(n),Q(n))−gravity  (6)

where transform_(v) is a standard vector transform that transforms Ai toa new direction according to Q.

At 510, the acceleration is integrated to determine a current velocityV(n) as follows:

V(n)=V(n−1)+Aw(n)*time_step  (7)

If left uncorrected, velocity V(n) will drift over time. Thus, astatistical model is used to remove the accelerometer's bias or zerooffset, and minimize errors in velocity estimation.

In some embodiments, the statistical model is a distribution model ofsensor output features to ground truth human motion velocity. The motiondata of various human users wearing various sensors and performingexercises is captured using a system that is independent of the sensorsand that is known to produce accurate measurements. For example, acamera-based system is used in some embodiments to capture sampleposition data of the users' leg movements. Multiple position data can becaptured for a single action such as a single step. To build the model,sample data of many steps taken by many different users wearingdifferent sensors is captured. The sample position data may be furtherprocessed to accurately derive other data such as velocity associatedwith these motions. In particular, the velocity that is determined usingthe independent system is referred to as “ground truth velocity.”Meanwhile, the sensors also capture data, and a set of sensor outputfeatures derived based on the sensors' outputs is determined. Examplesof the sensor output features include the statistical variance,frequency, and magnitude of the captured sensor data. The featurescharacterize the motions being performed. For example, a slow motiontends to correspond to a low variance in the sensor output data, and afast motion tends to correspond to a high variance in the sensor outputdata.

At 512, a probability distribution of the ground truth velocity for aspecific sensor output feature is determined based on a statisticalmodel for the given sensor output feature. Examples of a sensor outputfeature include frequency, variance, standard deviation, and any otherappropriate characteristics derived from the acceleration data, theangular velocity data, and/or magnetic field data. In some embodiments,the sensor output feature is a vector in multiple dimensions where eachdimension corresponds to a particular feature of a particular type ofsensor output (e.g., the frequency for acceleration data). Theprobability distribution function is expressed as P (Vg,sensor_feature).

FIG. 5B is a diagram illustrating an example statistical model of groundtruth velocities relative to a specific sensor output feature in asingle dimension. In this example, the sensor output feature correspondsto the frequency of one set of sensor measurement data (e.g., asdetermined by performing an operation such as a Fourier transform on thesensor measurement). Other sensor output features such as variance,standard deviation, etc., can be used and correspond to differentstatistical distribution functions. Specifically, the statistical modelin this example is represented as a 3-D surface 550 that corresponds tofunction P (Vg, sensor_feature). Given this distribution function, for agiven frequency, there is a corresponding probability distribution ofground truth velocities (e.g., function curve 552). Accordingly, foreach sample point, there is a corresponding sensor output feature value,and a corresponding distribution of possible human velocity, expressedas P_(x)′(Vg), where x is the sample index. Further, given theprobability distribution P, there is a corresponding preset alpha thatis set based on how stable the distribution is, which is used in theinterpolation function (6).

Returning to FIG. 5, at 514, an offset of the velocity is determinedsuch that the sum of probabilities for n over a sample window of (n0,n1) is maximized, as follows:

Maximize(Σ_(n=n0) ^(n1) P _(n)′(V(n)−Voff(n)))  (8)

where Voff(n)=a*n+b, where n is the sample index or time. Anoptimization algorithm such as a search-based optimization or any otherappropriate optimization algorithm is used to solve for a and b tomaximize the sum of the corrected probability values over the window n0and n1. The window size is preferably set to the average time for atypical user to perform a motion (e.g., if it takes on average 0.5seconds to perform a motion and the sample rate is 100 samples/second,then the window size is preferably set to 0.5*100=50 samples.)

At 516, the corrected current output velocity V″(n) is capped based onthe current output feature's distribution at the current sample pointand the offset as follows:

V″(n)=Normalize(V′(n))*Min(magnitude(V′(n)),Vmax_magnitude(n))   (9)

where V′(n)=V(n)−Voff, Vmax_magnitude(n) is the maximum velocity themotion can reach based on the current sensor output features with acertain degree of confidence or probability. For example, it is 99%likely that the user cannot reach more than two meters per second withthis motion pattern.

Although the example process above includes a magnetometer measurement,the magnetometer measurement is optional. A coordinate system can stillbe built without the magnetometer measurement, even though informationabout the North direction is lost and the horizontal direction cannot bedetermined. In practice, the coaching system can still give the userfeedback on his form by arbitrarily selecting a north direction.

The automated fitness coaching system is often implemented in a lossyenvironment where some samples transmitted from the sensor device to themanagement device can be lost. An index-based transmission technique isimplemented in some embodiments to make corrections in the event of dataloss. In some embodiments, each set of sensor outputs (A, G, andoptionally M) is transmitted with a sample index that increments from aninitial value to a maximum value (e.g., from 0 to 255) then resets tothe initial value, so that the receiver (e.g., the management device)can track and repair lost packets or samples. If one or more samples arelost, the sample points are interpolated based on the samples before andafter the lost ones such as the following:

S(lost)=Interpolate(S(lost−1),S(lost+1),0.5)  (10)

The above example interpolation function is a linear interpolationfunction. Different interpolation techniques can be used in otherembodiments.

In some embodiments, the management device is installed with differentapplications for different types of exercises. The user may select anappropriate application for the specific exercise he is doing. Theapplication determines, based on sensor measurements and using a processsuch as 500 described above, corrected core measurements for velocity,orientation, etc. In some embodiments, the corrected core measurementsare further processed to derive additional corrected core measurements(e.g., acceleration) and/or additional form measurements such ascadence, range of motion, landing impact, etc.

FIG. 6 is a flowchart illustrating an embodiment of a process fordetermining a form measurement associated with a single repetition of anexercise motion. Process 600 can be performed on an evaluation modulesuch as 204 of FIG. 2A.

At 602, a selected sequence of samples (e.g., a set of corrected coremeasurements output by the estimator) is used to determine a dominantrotational axis. In some embodiments, a singular vector decomposition(SVD) of the rotational vectors Rw associated with the sequence ofsamples is performed to determine the dominant rotational axis.

At 604, the angular velocity vectors of the samples are projected on thedominant rotational axis to identify samples that are involved in afirst rotational direction and samples that are involved in a second,different rotational direction relative to the dominant rotational axis.Specifically, for each sample point, a dot product of the rotationalvector and the dominant rotational axis is computed.

At 606, distinct parts of the motion are identified based on patternrecognition, using the results of step 604. For example, the forwardswing and the backward swing of the leg motion during running or the armduring swimming can be identified.

At 608, one or more form measurements such as cadence, range of motion,landing impact, etc., associated with a single repetition of a set ofexercise motions taken by the user is determined based at least in parton the recognized distinct parts. Details of the determination areexplained below in connection with FIGS. 7A and 7B.

FIG. 7A is a sample data diagram illustrating the data processing ofFIG. 6. In FIG. 7A, x-axis corresponds to time. Signal 702 correspondsto a projection of the angular velocity vectors on the dominantrotational axis according to 604 of process 600. Portions above thex-axis (e.g., the portion of signal 702 between markers 706 and 708) areinvolved in rotational motion in one direction, and portions below thex-axis (e.g., the portion of signal 702 between markers 704 and 706) areinvolved in rotational motions in the opposite direction.

A single repetition of an exercise can have multiple portionscorresponding to different patterns. For example, based on patternrecognition (606 of process 600), the portion of signal 702 betweenmarkers 704 and 706, which matches a backward swing pattern of a singlepeak, is deemed to correspond to a backward swing of the leg; theportion of signal 702 between markers 706 and 708, which matches aforward swing pattern of multiple peaks, is deemed to correspond to aforward swing of the leg. Any appropriate pattern recognition techniquecan be used. Different exercises may have different patterns that areempirically determined.

In some embodiments, the form measurements derived based on theestimator's outputs include the forward time, the backward time, and alanding time. In FIG. 7A, line 710 corresponds to acceleration data. Thelanding time is determined by recognizing a landing pattern in theacceleration data, specifically a spike 330 in the acceleration. Duringthe landing process the acceleration undergoes the greatest change. Thepoint of impact, 712, is identified as the time spike 300 reaches itspeak, and the landing time is identified as the time between 714 and716. The landing impact is computed as the acceleration at the point ofimpact.

In some embodiments, the form measurement derived based on theestimator's outputs includes a cadence measurement. Specifically, thetotal time between 704 and 708 corresponds to the time for a singlestep, and the inverse of this value corresponds to the instantaneouscadence of the step.

In some embodiments, the form measurement includes a range of motionmeasurement. In this case, the range of motion is computed as thedifference between the orientation measurement (Qw) at time 708 and time704. This measurement can also be used to evaluate the stride length orstride angle of the user.

FIG. 7B is a diagram illustrating a simulation of a user's leg motionduring a running exercise. A sensor device is attached to the legillustrated using dark lines 401-440. The leg motion at 420 correspondsto time 704 of FIG. 7A, and the leg motion at 430 corresponds to time708 of FIG. 7A.

In some embodiments, the form measurement includes an efficiencymeasurement. Specifically, an eigenfunction is applied to the angularvelocity in world space to obtain an eigenvalue. The energyconcentration of the resulting eigenvector indicates the energyconcentration and efficiency of the motion. In some embodiments, themost dominant component of the eigenvector is compared with the othercomponents to determine their difference. The greater the difference,the higher the efficiency. For example, the efficiency corresponding toan eigenvalue of (5, 1, 1) is higher than the efficiency correspondingto the eigenvalue of (5, 3, 1).

In some embodiments, the user wears multiple sensor devices, and thedata collected from these devices are processed and compared. Forexample, a runner may wear a sensor device on each leg. Measurementsfrom the devices are compared to determine the performance of each leg.In some embodiments, for each leg, the time, the range of motion, theefficiency, the average speed (measured as the range of motion/time),and/or the maximum speed during the forward swing phase, the backwardswing phase, and the landing phase are computed. The measurements arecompared to determine which side is stronger and/or more efficient.

In some embodiments, the form measurements are displayed to the user asfeedback. FIGS. 9A-9B are example user interface screens of showingfeedback. In this example, an application on the management devicedisplays the current stride rate (cadence) as the user is exercising inarea 902, the current rotational angle in area 904, and the currentspeed in area 906. The landing impact values are computed at 3-secondintervals, and are displayed in area 908. In this case, the landingimpact value (which corresponds to a corrected acceleration measurement)exceeds a threshold. In response, the automated fitness coach displays amessage indicating that the landing impact is higher than normal andoffers suggestions for reducing landing impact.

In some embodiments, the form measurement of the user is compared with atarget, and feedback is provided based on the comparison result. Forexample, the form measurements of a coached workout are preprogrammedand compared with the user's form data, and the user is informed of thecomparison result via audio, video, graphics, tactile feedback, etc. Insome embodiments, the coached workout is preprogrammed based on a coachperforming the workout and includes form measurements of the coach'smotions while performing the workout. For example, a boxing coach wearsa sensor device on each arm while performing a sequence of boxing moves.Audio and/or video information of the workout is captured using standardrecording equipment, and the form measurements of the workout aredetermined using the technique described above and recorded using amanagement device. The data is synchronized so that they can be playedback together.

When a user (also wearing a sensor device on each wrist) is ready toperform the workout, he activates a coaching application on themanagement device. A video of the recorded workout is played on themanagement device for the user to follow along. Meanwhile, the user'smotions are captured, and his corrected core measurements and/or formmeasurements are computed and compared to those of the recorded workoutby the coach. The user can be given feedback based on the comparisonresults. For example, if the velocity of the user's motion is less thanthat of the coach by some threshold, the user will be informed that “youneed to punch faster”; if the rotation of the user's motion is less thanthat of the coach by some other threshold, the user will be informedthat “you need to rotate your arm more”; if the form measurements of theuser closely resemble the recorded form measurements by the coach, hewill receive encouraging comments such as “good job, keep it up.” Anyother audio/visual/sensory queues can be provided as feedback. Invarious embodiments, the feedback messages are determined based on amapping of comparison results to feedback information, a rule-basedlookup, or the like.

FIG. 8 is an example user interface screen displaying a screenshot of acoached boxing workout. The video of coach 800 performing the workout isdisplayed. During the workout, the coach wears sensor devices 820 and860 on her wrists. The form measurements of motions by the coach's leftand right arms are visualized using two sets of lines 850. The linesadvance as the exercise progresses. Boxes 830 are used to indicate thepresent time. In this example, the lines continue to move up until theyare outside the designated area and no longer visible. In this case, forthe ease of visualization, a set of corrected core measurements and/orform measurement vectors are converted into a scalar value such that alonger line represents a faster and wider motion. In some embodiments,the magnitudes of the vectors are summed to generate the scalar value.Other types of representations can be used. As the user follows alongthe workout, his real-time data is processed. The form measurements ofthe user's movements can be translated into line lengths in the same wayas the form measurements of the coach's movements as lines 840 which areoverlaid on the set of lines 850. The left/right hand movement andattitude are represented by the visual object 880 and 890. Moreover, thespeed of the user's motion is also represented by circle 870 in such away that the greater the speed, the greater the size of the circle. Anevaluation of the user's performance is displayed in a status area 810.In some embodiments, the evaluation is based on an aggregate value ofthe differences between the user's form measurements and the coach'sform measurements.

In some embodiments, a feedback model is constructed using a machinelearning technique. FIGS. 10A-10B are block diagrams illustrating alearning machine. In FIG. 10A, a set of sample training data 1002 isinput into a learning machine 1000 during a training process. Each pieceof training data has a known set of inputs and a known result (alsoreferred to as an output) associated with the known set of inputs. Insome embodiments, the sample training data comprises core and/or formmeasurements of various users, and the outputs are the known results oftheir workouts (e.g., injury rate, fatigue level, recovery time,performance improvement based on a previous training session, etc.).Other training data can be used. The learning machine can be implementedusing a decision tree, a neural network, a Bayesian classifier, asupport vector machine (SVM), or any other appropriate machine learningmethods. The learning technique analyzes the training data and generatesan inferred function that models the input-output (result) relationshipfor the training data. Open source libraries and tools exist forimplementing such learning techniques. In other words, a model (thefunction inferred from the training data) is used to make a predictionof the output result given an input. In some cases, the predictedoutputs generated by the model are compared with the actual outputs ofthe samples to determine the prediction errors. The prediction errorsare fed back to the model to adapt the model. For example, thecoefficients or parameters of the model may be adjusted. The trainingdata (or a portion of the training data) is input to the adapted model,and errors of the model are measured again. Adaptation changes to themodel that improve the prediction (e.g., resulting in a smaller errorrate or magnitude) are accepted. The adaptation process may repeat untilan acceptable prediction error rate is reached. After training, in FIG.10B, a new input with no known output is sent to learned model 1000,which will generate a predicted output.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

What is claimed is:
 1. A motion data processing system, comprising: oneor more processors configured to: receive a plurality of motion datasamples pertaining to a plurality of motions performed by a user;determine, based at least in part on the plurality of motion datasamples, a set of one or more corrected core measurements correspondingto one or more body positions of the user during the plurality ofmotions, the set of one or more corrected core measurements comprisinginterpolated orientation data based at least in part on an interpolationof corrected orientation data and current orientation data obtained fromthe plurality of motion samples; and provide feedback to the user, thefeedback being based at least in part on a result of the comparing theone or more corrected core measurements to the set of reference motions;and one or more memories coupled to the one or more processors,configured to provide the one or more processors with instructions. 2.The system of claim 1, wherein the plurality of motion data samplesfurther comprises a plurality of magnetic field data samples.
 3. Thesystem of claim 1, wherein the set of one or more corrected coremeasurements is determined based at least in part on a statisticalmodel.
 4. The system of claim 1, wherein to determine the currentorientation data sample includes performing a quaternion transform of aprevious interpolated and corrected orientation data sample and acurrent angular velocity data sample.
 5. The system of claim 1, whereinthe plurality of data samples comprise a plurality of acceleration datasamples obtained by an accelerometer and a plurality of angular velocitydata samples obtained by a gyroscope.
 6. The system of claim 5, whereinthe plurality of acceleration data samples and the plurality of angularvelocity data samples are preprocessed, the preprocessing including todetect a lost data sample and to estimate the lost data sample byinterpolation of received data samples.
 7. The system of claim 1,wherein the set of one or more corrected core measurements furthercomprises corrected velocity data that is based at least in part onorientation data and a statistical model.
 8. The system of claim 7,wherein the statistical model is a distribution model of sensor outputfeatures to ground truth motion velocity.
 9. The system of claim 7,wherein the orientation data used to determine the velocity data is theinterpolated orientation data.
 10. The system of claim 7, wherein todetermine the corrected velocity data includes to: determine anacceleration in world space; integrate the acceleration to determine acurrent velocity data sample; determine a probability distribution of aground truth velocity based at least in part on the statistical model;determine an offset velocity to maximize a sum of probabilities for asample window; and determine a corrected current output velocity basedat least in part on the probability distribution and the offsetvelocity.
 11. The system of claim 1, wherein the one or more processorsare further to determine a form measurement that characterizes a form ofthe user during a subset of the plurality of motions.
 12. The system ofclaim 11, wherein the form measurement comprises a measurementpertaining to one or more of cadence, stride, range of motion,efficiency, speed, landing impact, and pronation.
 13. The system ofclaim 11, wherein to determine the form measurement includes to:determine a dominant rotational axis based at least in part on the setof the corrected core measurements; and identify a first set of samplesinvolved in a first rotational direction and a second set of samplesthat are involved in a second rotational direction based on a projectionon the dominant rotational axis.
 14. The system of claim 11, wherein todetermine the form measurement further includes to identify distinctparts of the set of corrected core measurements based on patternrecognition; and determine the form measurement based on the identifieddistinct parts.
 15. The system of claim 1, wherein the set of one ormore corrected core measurements includes a quaternion vector associatedwith a subset of the plurality motions and a velocity vector associatedwith the subset of the plurality motions.
 16. The system of claim 1,wherein the feedback to the user is provided on user interface.
 17. Thesystem of claim 16, wherein the feedback comprises a recommendation forthe user to adjust a form of an exercise.
 18. The system of claim 16,wherein the feedback comprises a difference between the plurality ofmotions performed by a user and the set of reference motions.
 19. Amethod comprising: receiving, by one or more processors, a plurality ofmotion data samples pertaining to a plurality of motions performed by auser; determining, based at least in part on the plurality of motiondata samples, a set of one or more corrected core measurementscorresponding to one or more body positions of the user during theplurality of motions, the set of one or more corrected core measurementscomprising interpolated orientation data based at least in part on aninterpolation of corrected orientation data and current orientation dataobtained from the plurality of motion samples; and providing, by the oneor more processors, feedback to the user, the feedback being based atleast in part on a result of the comparing the one or more correctedcore measurements to the set of reference motions.
 20. A computerprogram product for motion data processing, the computer program productbeing embodied in a tangible computer readable storage medium andcomprising computer instructions for: receiving, by one or moreprocessors, a plurality of motion data samples pertaining to a pluralityof motions performed by a user; determining, based at least in part onthe plurality of motion data samples, a set of one or more correctedcore measurements corresponding to one or more body positions of theuser during the plurality of motions, the set of one or more correctedcore measurements comprising interpolated orientation data based atleast in part on an interpolation of corrected orientation data andcurrent orientation data obtained from the plurality of motion samples;and providing, by the one or more processors, feedback to the user, thefeedback being based at least in part on a result of the comparing theone or more corrected core measurements to the set of reference motions.